Skip to content

fix(subscription): treat absent schema-evolved TAG columns as NULL and flatten logical chains in filter parser - #19346

Open
VGalaxies wants to merge 2 commits into
apache:masterfrom
VGalaxies:gh/fix-subscription-tagfilter-schema-null-and-logical-depth
Open

VGalaxies wants to merge 2 commits into
apache:masterfrom
VGalaxies:gh/fix-subscription-tagfilter-schema-null-and-logical-depth

Conversation

@VGalaxies

Copy link
Copy Markdown
Contributor

Summary

Fixes two correctness issues in table-model subscription tag/column filtering:

  1. Treat absent schema-evolved TAG columns in older Tablets as SQL NULL (TabletTagFilter):
    • After ALTER TABLE ... ADD COLUMN new_tag STRING TAG, historical TsFiles and pre-alter WAL Tablets do not include new_tag in their column schema.
    • Previously, TabletTagFilter#resolveReferencedTagColumns unconditionally threw TagFilterEvaluationException.schemaBinding("Referenced TAG column is missing") even when validateBinding had already proven against TsTable that new_tag is a valid TAG column.
    • When table schema binding is validated (schemaBound == true, covering both literal and bound wildcard topics), absent referenced TAG columns now evaluate as SQL NULL (handled by getTagValue when columnIndex == null, preserving SQL three-valued logic). Unbound matchers without a TsTable binding continue to fail closed.
  2. Flatten same-operator AND/OR chains in ColumnFilterParser:
    • ColumnFilterParser.AstBuilder#visitLogicalBinary previously constructed left-skewed binary trees via LogicalExpression.and(left, right) / .or(left, right), causing flat chains of >31 predicates (e.g., 40 OR terms) to exceed TagFilterValidator.MAX_AST_DEPTH = 32.
    • Same-operator LogicalExpression children are now flattened into a single n-ary LogicalExpression (matching how TagFilterValidator, TagFilterEvaluator, ColumnFilterValidator, and ColumnFilterEvaluator already iterate node.getTerms()), while preserving operator precedence ((a OR b) AND c) and still rejecting genuinely deep nesting (>32 levels of NOT (...)).

Concrete Before-and-After Examples

  • Schema evolution (ALTER TABLE weather ADD COLUMN new_tag STRING TAG) with a pre-alter Tablet(region, device, temperature):
    • Before: filters such as new_tag IS NULL or new_tag = "v1" OR region = "north" threw TagFilterEvaluationException: Referenced TAG column is missing.
    • After: new_tag evaluates as SQL NULL — new_tag IS NULL retains all rows, new_tag = "v1" and NOT (new_tag = "v1") drop all rows under 3VL, and new_tag = "v1" OR region = "north" retains "north" rows; unbound matchers still throw on missing columns.
  • Flat 40-term logical chain (region = "r0" OR region = "r1" OR ... OR region = "r39"):
    • Before: parsed as a left-skewed tree of depth 40 and rejected with Tag-filter AST depth exceeds maximum of 32.
    • After: parsed as a single LogicalExpression(OR, 40 terms) of depth 2 and evaluated accurately; 33-level nested NOT (NOT (...)) is still rejected.

Verification

  • mvn test -pl iotdb-core/datanode -am -Dsurefire.failIfNoSpecifiedTests=false -Dtest=TagFilterParserTest,TabletTagFilterTest,ColumnFilterParserTest,SubscriptionFilterSnapshotTest: 28/28 passed (7 + 8 + 9 + 4)
  • mvn test-compile -pl iotdb-core/datanode -am -DskipTests: exit 0
  • mvn test-compile -pl iotdb-core/datanode -am -P with-zh-locale -DskipTests: exit 0
  • mvn spotless:check -pl iotdb-core/datanode: exit 0

@VGalaxies
VGalaxies requested a balanced review from Copilot November 20, 2026 09:28
final Expression expression) {
if (expression instanceof LogicalExpression
&& ((LogicalExpression) expression).getOperator() == operator) {
terms.addAll(((LogicalExpression) expression).getTerms());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Avoid quadratic copying when flattening a chain. The grammar is left-recursive, so for a OR b OR ..., each parent reaches this branch with all prior terms in the left child. addAll recopies those terms for every predicate, and LogicalExpression copies the new list again. Parsing a length-N flat column filter therefore allocates O(N^2) references. The column-filter path has no size/depth cap, so a sufficiently large valid topic filter can cause a large CPU/heap spike while refreshing its matcher. Please collect same-operator terms in one pass or impose a limit.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this! Fixed in 7a9280d62e:

  • Replaced the bottom-up recursive appendLogicalTerms / addAll flattening in ColumnFilterParser.AstBuilder#visitLogicalBinary with an iterative single-pass walk using an explicit ArrayDeque<BooleanExpressionContext> stack (also peeling parenthesized same-operator sub-expressions via unwrapParenthesized).
  • Same-operator chains are now flattened into a single LogicalExpression in strict O(N) time and O(N) allocations without intermediate LogicalExpression copies or Java recursion depth growth.
  • Strengthened ColumnFilterParserTest#testFlattenedOrChainParsesIntoSingleLogicalExpression (500-term OR chain checking exact left-to-right order) and added testParenthesizedSameOperatorFlattensWhilePreservingMixedPrecedence.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues
What changed in this PR

Fixes correctness and scalability issues in subscription filter evaluation by handling schema-evolved missing TAG columns as SQL NULL (when schema binding is available) and by flattening logical AND/OR chains to avoid excessive AST depth.

Changes:

  • Treat missing, schema-evolved TAG columns in older Tablets as SQL NULL when table schema binding is present.
  • Flatten same-operator logical chains in ColumnFilterParser to prevent MAX_AST_DEPTH violations for wide AND/OR expressions.
  • Add regression tests for schema evolution behavior and flattened logical-chain parsing/evaluation.
File Description
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​subscription/​tagfilter/​TabletTagFilter.java Makes schema binding return a boolean and uses it to treat absent TAG columns as NULL instead of throwing.
iotdb-core/​datanode/​src/​main/​java/​org/​apache/​iotdb/​db/​subscription/​columnfilter/​ColumnFilterParser.java Flattens AND/OR logical chains into n-ary LogicalExpression terms to reduce AST depth.
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​subscription/​tagfilter/​TagFilterParserTest.java Adds tests ensuring flat logical chains don’t exceed max AST depth and precedence is preserved.
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​subscription/​tagfilter/​TabletTagFilterTest.java Adds tests for schema-evolved missing TAG columns treated as NULL when schema-bound.
iotdb-core/​datanode/​src/​test/​java/​org/​apache/​iotdb/​db/​subscription/​columnfilter/​ColumnFilterParserTest.java Adds test validating flattened OR chains parse/evaluate as a single LogicalExpression.

Review effort: Lite

Comment on lines +211 to +218
final LogicalExpression.Operator operator =
Objects.nonNull(context.AND())
? LogicalExpression.Operator.AND
: LogicalExpression.Operator.OR;
final List<Expression> terms = new ArrayList<>();
appendLogicalTerms(terms, operator, visit(context.booleanExpression(0)));
appendLogicalTerms(terms, operator, visit(context.booleanExpression(1)));
return new LogicalExpression(operator, terms);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note. For the tag-filter path, TagFilterValidator already enforces MAX_AST_NODES = 256 across all flattened terms (visitLogicalExpression invokes process() -> accountNode() on every element in node.getTerms()), so wide AND/OR chains remain strictly bounded by MAX_AST_NODES. ColumnFilterValidator did not enforce MAX_AST_DEPTH/MAX_AST_NODES previously, and the quadratic flattening cost in ColumnFilterParser has now been eliminated in 7a9280d62e.

Comment on lines +215 to +230
final List<Expression> terms = new ArrayList<>();
appendLogicalTerms(terms, operator, visit(context.booleanExpression(0)));
appendLogicalTerms(terms, operator, visit(context.booleanExpression(1)));
return new LogicalExpression(operator, terms);
}

private static void appendLogicalTerms(
final List<Expression> terms,
final LogicalExpression.Operator operator,
final Expression expression) {
if (expression instanceof LogicalExpression
&& ((LogicalExpression) expression).getOperator() == operator) {
terms.addAll(((LogicalExpression) expression).getTerms());
} else {
terms.add(expression);
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7a9280d62e: ColumnFilterParser.AstBuilder#visitLogicalBinary now walks same-operator LogicalBinaryContext subtrees iteratively with an explicit ArrayDeque stack and collects all terms into a single LogicalExpression in O(N) time without quadratic addAll copying.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants